Skip to content

Gate output-centric image_to_nchw staging to discrete GPUs (#21022)#21022

Merged
meta-codesync[bot] merged 1 commit into
pytorch:mainfrom
SS-JIA:export-D112599710
Jul 21, 2026
Merged

Gate output-centric image_to_nchw staging to discrete GPUs (#21022)#21022
meta-codesync[bot] merged 1 commit into
pytorch:mainfrom
SS-JIA:export-D112599710

Conversation

@SS-JIA

@SS-JIA SS-JIA commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary:

The image_to_nchw output-staging shader can dispatch one thread per NCHW
output element so writes to the host-visible staging buffer are fully coalesced.
On a discrete GPU the staging buffer is read back over PCIe, where coalescing
writes is a large win: ~11x end-to-end on an RTX 4080 laptop for an ESPCN 2x
super-resolution model.

That output-centric scheme fetches each texel up to 4 times (once per
component). On integrated (mobile / on-die) GPUs the staging buffer is not
PCIe-backed, so write coalescing buys nothing and the redundant fetches are a
net loss: measured +15-42% image_to_nchw per-dispatch on Mali-G715 and +3-6%
on Adreno 750.

Gate the two strategies on the physical device type via
Adapter::is_integrated_gpu():

  • discrete (not integrated) -> image_to_nchw_coalesced_* (output-centric)
  • integrated (mobile) -> image_to_nchw_* texel-centric (one thread per texel,
    single fetch, contiguous NCHW writes), which is the default.

Gate on device type, NOT has_unified_memory(): a discrete GPU with Resizable
BAR exposes a DEVICE_LOCAL | HOST_VISIBLE memory type, so has_unified_memory()
returns true for it and would wrongly route it to the slow texel-centric path
(measured ~10x end-to-end regression on an RTX 4080 laptop).

image_to_nchw.glsl emits both variants via a COALESCED_WRITES codegen flag
(default False). get_tensor_to_nchw_shader selects the coalesced variant when
!is_integrated_gpu(); the global-workgroup-size picker keys off the shader
name so the dispatch topology matches the chosen variant.

Reviewed By: metascroy

Differential Revision: D112599710

@pytorch-bot

pytorch-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21022

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 5605479 with merge base 21554e5 (image):

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 17, 2026
@meta-codesync

meta-codesync Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@SS-JIA has exported this pull request. If you are a Meta employee, you can view the originating Diff in D112599710.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@meta-codesync meta-codesync Bot changed the title Gate output-centric image_to_nchw staging to discrete GPUs Gate output-centric image_to_nchw staging to discrete GPUs (#21022) Jul 20, 2026
@SS-JIA
SS-JIA force-pushed the export-D112599710 branch from 37460f1 to 0fbb318 Compare July 20, 2026 21:56
SS-JIA added a commit to SS-JIA/executorch-1 that referenced this pull request Jul 20, 2026
…1022)

Summary:

The `image_to_nchw` output-staging shader can dispatch one thread per NCHW
output element so writes to the host-visible staging buffer are fully coalesced.
On a discrete GPU the staging buffer is host-visible but not device-local
(PCIe-backed system RAM), where coalescing writes is a large win: ~11x
end-to-end on an RTX 4080 laptop for an ESPCN 2x super-resolution model.

That output-centric scheme fetches each texel up to 4 times (once per
component). On unified-memory (mobile) GPUs the staging buffer is not
PCIe-backed, so write coalescing buys nothing and the redundant fetches are a
net loss: measured +15-42% `image_to_nchw` per-dispatch on Mali-G715 and +3-6%
on Adreno 750.

Gate the two strategies on `Adapter::has_unified_memory()`:
- discrete (no unified memory) -> `image_to_nchw_coalesced_*` (output-centric)
- unified memory (mobile) -> `image_to_nchw_*` texel-centric (one thread per
  texel, single fetch, contiguous NCHW writes), which is the default.

`image_to_nchw.glsl` emits both variants via a `COALESCED_WRITES` codegen flag
(default False). `get_tensor_to_nchw_shader` selects the coalesced variant when
`!has_unified_memory()`; the global-workgroup-size picker keys off the shader
name so the dispatch topology matches the chosen variant.

Reviewed By: metascroy

Differential Revision: D112599710
…1022)

Summary:

The `image_to_nchw` output-staging shader can dispatch one thread per NCHW
output element so writes to the host-visible staging buffer are fully coalesced.
On a discrete GPU the staging buffer is read back over PCIe, where coalescing
writes is a large win: ~11x end-to-end on an RTX 4080 laptop for an ESPCN 2x
super-resolution model.

That output-centric scheme fetches each texel up to 4 times (once per
component). On integrated (mobile / on-die) GPUs the staging buffer is not
PCIe-backed, so write coalescing buys nothing and the redundant fetches are a
net loss: measured +15-42% `image_to_nchw` per-dispatch on Mali-G715 and +3-6%
on Adreno 750.

Gate the two strategies on the physical device type via
`Adapter::is_integrated_gpu()`:
- discrete (not integrated) -> `image_to_nchw_coalesced_*` (output-centric)
- integrated (mobile) -> `image_to_nchw_*` texel-centric (one thread per texel,
  single fetch, contiguous NCHW writes), which is the default.

Gate on device type, NOT `has_unified_memory()`: a discrete GPU with Resizable
BAR exposes a DEVICE_LOCAL | HOST_VISIBLE memory type, so `has_unified_memory()`
returns true for it and would wrongly route it to the slow texel-centric path
(measured ~10x end-to-end regression on an RTX 4080 laptop).

`image_to_nchw.glsl` emits both variants via a `COALESCED_WRITES` codegen flag
(default False). `get_tensor_to_nchw_shader` selects the coalesced variant when
`!is_integrated_gpu()`; the global-workgroup-size picker keys off the shader
name so the dispatch topology matches the chosen variant.

Reviewed By: metascroy

Differential Revision: D112599710
@SS-JIA
SS-JIA force-pushed the export-D112599710 branch from 0fbb318 to 5605479 Compare July 20, 2026 22:08
@meta-codesync
meta-codesync Bot merged commit 7f5c264 into pytorch:main Jul 21, 2026
189 of 191 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants